Bayesian Identi cation of Outliers in Computerized Adaptive Tests

We consider the problem of identifying examinees with aberrant response patterns in a computerized adaptive test (CAT). The vector of responses yi of person i from the CAT comprise a multivariate response vector. Multivariate observations may be outlying in many di erent directions and we characterize speci c directions as corresponding to outliers with di erent interpretations. We develop a class of outlier statistics to identify di erent types of outliers based on a control chart type methodology. The outlier methodology is adaptable to general longitudinal discrete data structures. We consider several procedures to judge how extreme a particular outlier is. Data from the National Council Licensure EXamination (NCLEX) motivates our development and is used to illustrate the results. Eric T. Bradlow is Assistant Professor of Marketing and Statistics, Wharton School of Business, University of Pennsylvania, Philadelphia, PA 19104-6371. Part of this work was performed while he was Associate Research Scientist, Statistics and Psychometric Research Group, Educational Testing Service, Princeton, NJ 08541-0001. Robert E. Weiss is Assistant Professor, Department of Biostatistics, UCLA School of Public Health, Los Angeles, CA 90095-1772. Meehyung Cho is Graduate Student, Department of Biostatistics, UCLA School of Public Health, Los Angeles, CA 90095-1772. This work was supported by grants #GM50011 from the National Institute of General Medical Sciences of the National Institutes of Health and NCSBN/ETS JRC #415-83. The authors thank Walter D. Way of the Chauncey Group International for his guidance on de ning interesting outlier types.